Repository navigation
[feature](cloud) Support declaring compute_group on routine load and async MV - #69749
Conversation
…async MV ### What problem does this PR solve? Issue Number: None Related PR: apache#67010 Problem Summary: branch-4.1 (apache#67010, also in branch-4.2) lets a background job be pinned to a compute group; master has none of it. Without it a routine load job silently snapshots whatever compute group the creating session was on, and an async materialized view has no compute group property at all: automatic refreshes run wherever `admin` resolves to, while a manual REFRESH borrows the triggering session's group, so the same MV can refresh in two different places depending on who triggered it. This adds a `compute_group` property to CREATE / ALTER ROUTINE LOAD and to async materialized views, with the same semantics as branch-4.x: - The declaration lives in the existing property maps (`RoutineLoadJob.jobProperties`, `MTMV.mvProperties`) and is written only when declared, so there is no new persisted field, journal type or meta version, and an absent key keeps today's implicit resolution. - `DEFAULT` is rejected (reserved for "follow the owner's default group" in the later binding model); non-cloud mode rejects the property, and `ConnectContext.setCloudCluster()` becomes a no-op outside cloud mode. - Routine load resolves the effective cluster in `RoutineLoadJob.getCloudCluster()`, which backend selection, the task sent to BE, the workload group namespace, the plan context and SHOW ROUTINE LOAD all go through, so Kafka and Kinesis jobs both follow it. CREATE now validates the workload group in the declared group's namespace, and re-parsing the stored statement at metadata load does not re-validate the declared group (a failure there would cancel the job for good). - An async MV resolves the declared group before the session's group, on the planning context and on every refresh context. - The declared group is re-checked before each task: in `RoutineLoadTaskScheduler`, before backend allocation and beginTxn (a dropped group pauses the job retryably, a revoked USAGE pauses it with CANNOT_RESUME_ERR), and in both MV refresh paths (the refresh task fails). Differences from branch-4.x, from master's structure: master's MTMVTask splits refresh into the IVM incremental path and the partition / complete path, so `applyComputeGroup()` is factored out of `setupComputeGroup()` and the re-check runs in both paths; master's routine load also has a Kinesis source, covered by the same shared code. The 4.x unit test for a task without a task context is not carried over: MTMVJob always creates tasks with one, and `setupComputeGroup()` already relies on it. The processlist test of apache#68111 now sets the session variable directly, as branch-4.x did when it picked apache#68111 (apache#68948), because `ConnectContext.setCloudCluster()` is a no-op outside cloud mode. The ported tests use JUnit 5, and MTMVComputeGroupTest gains two cases for master's `setupComputeGroup()` and the MV re-check. ### Release note Support declaring `compute_group` on routine load jobs and async materialized views in cloud mode, so a background job can be pinned to a specific compute group instead of inheriting one implicitly. ### Check List (For Author) - Test: Unit Test (RoutineLoadComputeGroupTest, MTMVComputeGroupTest, ComputeGroupBindingUtilTest and the existing tests of the changed and dependent classes); the three regression suites need a cloud cluster (and Kafka) and were not run locally - Behavior changed: Yes, only for jobs that declare `compute_group`: such an MV refreshes in its declared group even for a manual REFRESH from another group, and such a job fails its next task when the group is dropped or its owner's USAGE is revoked - Does this need documentation: Yes, the new `compute_group` property of routine load and async MV (cloud mode only, `DEFAULT` reserved) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of all 20 changed files at 124917f3e36cf3d5646ffcd0d0720ef631764325. Five verified issues require changes: routine-load metadata replay can permanently cancel a recoverable job (P1); a suspended declared group is not auto-started before Kafka metadata selection, combined group/offset ALTER uses the old group, MV query hints override the declaration, and concurrent MV ALTER can mix compute/workload groups (four P2s). All five are reported inline. There were no existing inline findings to carry forward and no additional user-provided focus.
Required checkpoint conclusions:
- Goal and proof: basic explicit pinning, undeclared fallback and task validation are implemented; the five lifecycle/execution cases prevent the complete intended behavior. New unit/regression tests cover basic branches; they were inspected only.
- Scope and reuse: focused property/getter/context changes reuse a shared validator and existing metadata maps. The broader non-cloud setter change has no verified production regression after caller review.
- Concurrency and locks: scheduler, SQL ALTER and refresh-worker interactions were traced. Existing routine-load locking/abandoned-task checks remain; MV paired-property reads are inconsistent under concurrent ALTER (inline). No new lock-order inversion or heavy work under a newly added lock was found.
- Lifecycle: task renewal/replanning and context cleanup paths were checked. Replay cancellation and suspended-target activation are inline issues. No new threads, ownership cycles or C++ static-initialization concerns apply.
- Configuration: compute_group is an object property, ALTERable through existing paths and reread without restart. No global configuration was added; MV paired-property observation needs a consistent snapshot.
- Compatibility: declarations use existing serialized maps/journal operations, with absent keys preserving fallback. No binary/storage-layout change was introduced; new behavior requires an FE that understands the property.
- Thrift/protobuf: no definitions, field IDs or enum values changed; release-branch ID comparison is not applicable.
- Parallel paths: CREATE/ALTER/replay, Kafka single/multi-table routing, manual/automatic MV refresh, IVM/partition/complete/retry/no-work paths, and cloud/non-cloud behavior were traced. Kinesis rejects cloud mode upstream. Pre-apply ALTER conversion and hint reapplication are missed consumers reported inline.
- Special conditions: empty values, cloud-only support, reserved DEFAULT, legacy null owner, and retryable missing-group versus non-resumable revoked-USAGE errors were reviewed. Virtual names are present in the same map used by USE; no distinct name-resolution bug was found.
- Test coverage: new tests exercise validation, fallback, backend isolation, reporting and task rejection. Missing scenarios match the five findings; incremental execution and concurrent ALTER lack end-to-end coverage. A repeated AUTO assertion may inspect old FIFO task history, weakening that check.
- Test results: no result files changed. No builds, tests, benchmarks or runtime validation were run under this static-only contract; author-reported test success was not independently reproduced.
- Observability: group/user errors and task reporting are present. Hint override can report A while executing in B; suspended/old-group metadata failures obscure the usable target. No additional critical metric gap was substantiated.
- Persistence/failover: existing CREATE/ALTER write/replay mechanisms carry the key. Routine-load reconstruction still resolves the original declared group through workload-group validation and can set final CANCELLED (inline). MV map replay adds no live validation.
- Writes/transactions: the routine-load recheck precedes backend allocation and beginTxn, and existing commit/visibility/offset rules remain. No new partial-commit, row-visibility, delete-bitmap or memory-ownership defect was verified; routing and replay failures are reported.
- FE-BE propagation: existing task cloud_cluster/context fields carry the effective value; no new transport field was added. MV hint mutation can change the value actually consumed by the coordinator (inline).
- Performance: helper checks use local privilege lookup and a sorted cluster-name snapshot per task. No substantiated high-impact performance regression or new validation RPC was found.
- Other concerns and limits: raw virtual-name mismatch, unsupported non-cloud setter regression, speculative context leaks and deterministic AUTO-test failure were dismissed with evidence. Review scope is complete; execution and timing remain unvalidated by design.
Review converged after 3 round(s).
| // against the compute group the job will actually run in, not the session's one. | ||
| tmpCtx.setCloudCluster(StringUtils.isEmpty(this.computeGroupName) | ||
| ? ConnectContext.get().getCloudCluster() | ||
| : this.computeGroupName); |
There was a problem hiding this comment.
[P1] Skip workload-group resource resolution during metadata replay
For a job created with both compute_group='cg_a' and workload_group='wg_a', setReplay(true) skips the direct compute-group check but still reaches this lookup with cg_a. If that group has been dropped, renamed or scaled to zero, getWorkloadGroup(tmpCtx) calls ComputeGroupMgr.getComputeGroupByName and throws. RoutineLoadJob.gsonPostProcess() catches it and sets the job to final CANCELLED, so bringing the group back cannot recover it with RESUME. This also reparses the original group after an ALTER to another group. Honor replay mode for the downstream resource lookup too; the new replay test only supplies compute_group and misses this branch.
There was a problem hiding this comment.
Confirmed, and fixed in 4ed5856. On the metadata load path setReplay(true) skipped the compute group check, but the workload group existence check right after it still resolved the declared group (WorkloadGroupMgr.getWorkloadGroup -> ConnectContext.getComputeGroup() -> ComputeGroupMgr.getComputeGroupByName), which throws once that group is dropped, renamed or scaled to zero, and gsonPostProcess turns the throw into the final CANCELLED. It is easier to hit than a dropped group alone: origStmt keeps the CREATE-time compute_group, so a job moved to another group by ALTER was cancelled by the next restart after its old group was dropped.
The same lookup also failed on image load without any compute group. PersistMetaModules reads routineLoadJobs before workloadGroups, and until Env.loadWorkloadGroups runs the Env holds the empty WorkloadGroupMgr its constructor created, so a job created with workload_group could not find its group there (Can not find workload group wg_a in compute group default) and was cancelled by the first restart that loaded it from an image. That part predates this PR.
The re-parse only rebuilds the RoutineLoadDesc and never uses the resolved workload group, so the replay path now adopts both names without resolving either; CREATE and ALTER still validate them. testMetadataLoadDoesNotResolveWorkloadGroupInDeclaredComputeGroup replays a statement that declares a dropped compute group together with a workload group, and testImageLoadAdoptsWorkloadGroupBeforeWorkloadGroupsAreLoaded replays one with only a workload group against an empty WorkloadGroupMgr, as on image load; both fail without the change. branch-4.x has the same code.
| // keeps the existing implicit resolution, which is also what later versions rely on to | ||
| // tell a pinned group apart from an inherited one. | ||
| if (!StringUtils.isEmpty(info.getComputeGroupName())) { | ||
| jobProperties.put(COMPUTE_GROUP, info.getComputeGroupName()); |
There was a problem hiding this comment.
[P2] Wake the declared group before the first Kafka metadata request
With cloud auto-start enabled, a session on live cg_a can declare an existing but SUSPENDED cg_b. Validation accepts its name, and this property immediately makes Kafka metadata requests use B, but none of that path calls waitForAutoStart(B) as USE @B does. A CREATE with custom partitions therefore fails during its initial Kafka metadata/offset lookup with no live backends; otherwise the scheduler's partition lookup pauses the job and repeatedly retries without ever requesting B's resume. Activate the declared target before the metadata/backend lookup, without changing the caller's session group or waiting under the job lock.
There was a problem hiding this comment.
Confirmed for CREATE and ALTER, and fixed in 4ed5856. After the USAGE and existence checks, the declared group is now started with CloudSystemInfoService.waitForAutoStart (ComputeGroupBindingUtil.activateDeclaredComputeGroup), the third step USE @<group> takes in CloudEnv.changeCloudCluster. It runs on the statement thread before the job lock or any Kafka metadata request, does not change the caller's session group, and is not called on the metadata load path. It is not part of validateDeclaredComputeGroup, which async MVs share: an MV sends nothing to its group at CREATE, and its refresh resolves backends through getBackendsByCurrentCluster, which already starts the group it binds.
The scheduler half is not specific to declared groups. CloudRoutineLoadManager.getAvailableBackendIds and KafkaUtil.getBackendIdsForMetaRequest resolve backends by name and never start a suspended group on master, so a job whose implicit group is suspended after creation already pauses and retries without resuming it. Changing that affects every cloud routine load job and is out of scope for this port. testCreateWakesDeclaredComputeGroup and testAlterWakesDeclaredComputeGroup cover the two call sites, and the metadata load test checks that replay starts nothing.
| // Unlike workload group, the compute group can be fully validated right here, so do | ||
| // it now instead of letting a bad name pause the whole job on the next task. | ||
| ComputeGroupBindingUtil.validateDeclaredComputeGroup(ctx, computeGroup); | ||
| analyzedJobProperties.put(CreateRoutineLoadInfo.COMPUTE_GROUP, computeGroup); |
There was a problem hiding this comment.
[P2] Use the requested group when converting offsets during ALTER
A combined ALTER ROUTINE LOAD ... PROPERTIES('compute_group'='live_b') FROM KAFKA('kafka_partitions'='0', 'kafka_offsets'='OFFSET_END') validates B here, but KafkaRoutineLoadJob.modifyProperties() calls convertOffset() before installing these properties. That conversion obtains the current job's old group and sends its metadata RPC there. If the old group has disappeared or has no live backends, ALTER fails before rebinding even though B is usable. Timestamp offsets have the same issue. Pass the validated requested group into offset conversion, retaining the old group only when this ALTER supplies no new binding.
There was a problem hiding this comment.
Confirmed, and fixed in 4ed5856. KafkaRoutineLoadJob.modifyProperties converts the requested offsets before modifyPropertiesInternal installs the new properties, so both the getRealOffsets and the getOffsetsForTimes conversion went to the job's current group. It now passes the group this ALTER declares, and keeps the current group only when the ALTER declares none. testCombinedAlterConvertsOffsetsInTheGroupItDeclares alters a job bound to a dropped group to a live one together with kafka_offsets, and checks that an ALTER without compute_group still converts in the job's group.
| String effective = mtmv.getComputeGroup().orElse(taskContext.getComputeGroup()); | ||
| if (!Strings.isNullOrEmpty(effective)) { | ||
| // A no-op outside cloud mode. | ||
| ctx.setCloudCluster(effective); |
There was a problem hiding this comment.
[P2] Keep the declared group authoritative after query hints run
An MV with PROPERTIES('compute_group'='cg_a') AS SELECT /*+ SET_VAR(compute_group='cg_b') */ k1 FROM base still refreshes in B. For the partition path, the plan is InsertOverwrite -> LogicalSelectHint(SET_VAR(cg_b)) -> Project -> Scan: setup records/checks A, then EliminateLogicalSelectHint reapplies the hint during command analysis and overwrites the session field consumed by CloudCoordinator.prepare(). In the IVM path, parsing the query after setup also overwrites it. The task reports A although execution uses B. Reject conflicting hints or preserve the binding independently of hint-mutated session state, and validate/report the group the coordinator actually consumes.
There was a problem hiding this comment.
This is the existing precedence between a statement hint and an object's properties rather than something this PR introduces. compute_group is an alias of the cloud_cluster session variable, and a SET_VAR hint in the MV's own query is applied during analysis (EliminateLogicalSelectHint -> setVarOnceInSql), after the task has prepared its context. On master the same hint already overrides the group of an MV that declares none, and workload_group behaves the same way: MTMVPlanUtil.createMTMVContext copies the MV property into the session, and a SET_VAR(workload_group=...) in the query overrides it while SHOW keeps reporting the property; the compute group the task records is the declared one in the same way. The hint is part of the MV definition its owner wrote, so the refresh following it is consistent with every other session variable. Making the declaration win, or rejecting conflicting hints at CREATE, would be a new rule for hints in MV SQL that should cover workload_group too, which is outside this port. branch-4.x behaves the same.
| // The planning done on this context (base table resolution, partition calculation) must see | ||
| // the same compute group the refresh will execute in, otherwise the workload group would be | ||
| // looked up in a different namespace. | ||
| applyComputeGroup(ctx); |
There was a problem hiding this comment.
[P2] Read compute and workload groups as one binding snapshot
createMTMVContext() already reads the MV's workload group, while this later call independently reads its compute group. The refresh worker can read wg_a, then a SQL thread atomically ALTER both properties from (cg_a, wg_a) to (cg_b, wg_b), then the worker reads cg_b. If wg_a exists only in A, the ensuing workload-group lookup fails in B despite both stored configurations being valid. The task's MTMVJob lock does not serialize property ALTER, which only takes the MV lock. Both execution context paths have the same separated reads. Snapshot the related properties under one MV read lock and use that snapshot for context setup and checking.
There was a problem hiding this comment.
The window is real but very narrow, and its outcome is one failed refresh with an explicit error rather than a wrong result: an ALTER that changes both properties would have to commit between createMTMVContext reading the workload group and applyComputeGroup reading the compute group, a few statements apart on the refresh thread. That refresh then fails its workload group lookup in the new group (Can not find workload group ... in compute group ...), and the next one reads a consistent pair. branch-4.x has the same two reads, and this port keeps them: reading both properties under one MV read lock would close the window, but with no wrong result at stake we would rather not diverge from 4.x here.
TPC-H: Total hot run time: 28809 ms |
TPC-DS: Total hot run time: 151306 ms |
ClickBench: Total hot run time: 24.54 s |
FE UT Coverage ReportIncrement line coverage |
…on and offset changes ### What problem does this PR solve? Issue Number: None Related PR: apache#67010 Problem Summary: Three gaps in the routine load `compute_group` declaration, found in review; branch-4.x has the same code. 1. Metadata load (`RoutineLoadJob#gsonPostProcess`) re-parses the CREATE statement with `setReplay(true)`, which skipped the compute group check but not the workload group lookup right after it. That lookup resolves the workload group in the declared compute group, which throws once the group is dropped, renamed or scaled to zero, and `gsonPostProcess` turns the throw into the final CANCELLED. The stored statement is the CREATE one, so a job ALTERed from cg_a to cg_b was cancelled by the next restart after cg_a was dropped. The lookup also fails on image load without any compute group: `PersistMetaModules` reads `routineLoadJobs` before `workloadGroups`, so the Env still holds the empty `WorkloadGroupMgr` its constructor created, and a job created with `workload_group` is cancelled by the first restart that loads it from an image (shown by a unit test with an empty manager; not reproduced on a cluster). The replay path now adopts the workload group name without resolving it, as it already did for the compute group; CREATE and ALTER still validate both. 2. A declared compute group got the USAGE and existence checks of `USE @<group>` but was not started the way `USE @` starts a suspended group. CREATE / ALTER with custom Kafka partitions or offsets send their metadata request through the declared group's backends while the statement runs, so a suspended group made them fail with no live backend. They now call `waitForAutoStart` on it (`ComputeGroupBindingUtil#activateDeclaredComputeGroup`) on the statement thread, before any job lock and without changing the session's group. The scheduler resolving backends by name without starting a suspended group is unchanged; that is master's behavior for every cloud routine load job. 3. In one ALTER that sets `compute_group` and `kafka_offsets`, `KafkaRoutineLoadJob#modifyProperties` converted the offsets before installing the new properties, so the conversion went to the job's old group and failed when that group was gone. It now uses the group the ALTER declares, and the current group only when the ALTER declares none. ### Release note None ### Check List (For Author) - Test: Unit Test - RoutineLoadComputeGroupTest gains 5 tests, all red before this change; 24 related FE test classes, 398 tests pass; fe-core checkstyle passes - Behavior changed: Yes. Metadata load no longer resolves the workload group of a routine load job; CREATE / ALTER ROUTINE LOAD start a suspended declared compute group; a combined ALTER converts its offsets in the group it declares - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the first review round in 4ed5856.
RoutineLoadComputeGroupTest gains five tests, each red without its fix; 24 related FE test classes, 398 tests, pass with Checkstyle. |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of all 21 changed files at 4ed5856 found two additional P2 defects: routine-load SHOW CREATE loses the declared binding, and incremental MV dry runs bypass it. No new P0/P1 finding was identified.
Existing threads: 4399202162 (replay workload-group lookup), 4399202170 (CREATE/ALTER activation), and 4399202175 (combined ALTER offset conversion) are fixed at this head. The former P1 is therefore not retained as blocking. Existing MV hint-precedence and compute/workload snapshot threads 4399202181/4399202183 were revalidated with the author replies and are not duplicated; the narrow snapshot window remains present, and hint precedence follows the existing session-variable policy.
Required checkpoint conclusions:
- Goal and evidence: normal routine-load and automatic/manual MV task paths use the explicit group; inspected unit/regression tests cover validation, fallback, placement, replay, activation, offset conversion and runtime refusal. The two inline findings identify missing consumers.
- Scope and reuse: focused FE property plumbing, with shared validation and existing property maps. Sharing MV context initialization would also cover foreground execution.
- Concurrency and locks: traced SQL ALTER, job/task schedulers and transaction callbacks. Routine-load PAUSE clears tasks and ALTER is PAUSED-only; backend affinity must belong to the current group. Activation occurs before job locks. MV property ALTER uses its own lock, leaving the already-reported snapshot window. No additional deadlock was found.
- Lifecycle: metadata replay adopts resource names without resolving or activating them, and original CREATE parsing only rebuilds descriptors. Missing groups remain retryable; revoked routine-load USAGE prevents automatic resume. No new static initialization issue was found.
- Configuration: no global configuration added; compute_group is an object property read on subsequent work. Routine-load ALTER requires PAUSED; MV properties are reread without restart.
- Compatibility: existing Gson maps and alteration records carry the declaration; absent keys preserve prior implicit behavior. No new wire or structural metadata field. Mixed-binary execution was not exercised.
- Thrift/protobuf: no definitions changed; field-ID comparison is inapplicable.
- Parallel paths: checked Kafka custom/default/timestamp offsets, single/multitable planning, MV IVM and partition/complete paths, and the foreground dry-run omission. Kinesis currently rejects cloud creation, so no cloud Kinesis execution is claimed.
- Conditions and errors: cloud-only, empty, reserved DEFAULT, replay and legacy-owner branches have explicit rationale. Runtime checks use explicit identities without needing a background thread-local context.
- Coverage: added Java and regression tests were inspected. No cross-group dry-run or routine-load SHOW CREATE round-trip case covers the two new findings. The same-value MV ALTER test uses AUTO without changing base data and reads unsorted history, so it does not independently establish post-ALTER execution placement.
- Expected results: no result snapshots changed; assertions were reviewed statically. No builds, tests, benchmarks or cluster operations were run.
- Observability: effective groups appear in SHOW ROUTINE LOAD and MV task rows; failures identify the group. SHOW CREATE is incomplete as reported. Earlier routine-load partition discovery can still produce the existing metadata/backend diagnostic before the task check; this was not treated as a distinct new correctness defect.
- Persistence and failover: existing create/alter journals and image serialization preserve property maps; replay does not overwrite altered binding with original CREATE properties. No new replay resource dependency was found.
- Data writes and atomicity: new routine-load refusal precedes backend allocation and beginTxn; existing task-membership commit fences remain. MV INSERT/INSERT OVERWRITE transaction and snapshot logic is unchanged. No visibility/version/delete-bitmap regression was identified.
- FE/BE propagation: existing cloud_cluster task fields and coordinator session values carry the effective group; no new transmitted variable is required.
- Performance: checks add property reads, cached-name enumeration and privilege lookup. No substantiated material performance regression or new heavy operation under a lock was found; no performance measurements were taken.
- Other concerns and limits: all changed files, risk focuses and candidate dispositions were swept again. No additional user focus was supplied. Conclusions are static; cloud auto-start, real restart and end-to-end regression execution remain unexercised.
Review converged after 2 round(s).
| // keeps the existing implicit resolution, which is also what later versions rely on to | ||
| // tell a pinned group apart from an inherited one. | ||
| if (!StringUtils.isEmpty(info.getComputeGroupName())) { | ||
| jobProperties.put(COMPUTE_GROUP, info.getComputeGroupName()); |
There was a problem hiding this comment.
[P2] Preserve the declared group in SHOW CREATE ROUTINE LOAD
The new binding is stored here, but getShowCreateInfo() emits a fixed property list that never includes compute_group. SHOW CREATE ROUTINE LOAD calls that method, so creating or ALTERing a job to cg_b, exporting its DDL, and recreating it from a session on cg_a silently creates an unpinned job running in A. Emit the current getDeclaredComputeGroup() in that SQL when nonempty, preserving the absent-key behavior for jobs without a declaration, and cover the round trip after ALTER.
There was a problem hiding this comment.
Confirmed, and fixed in 7611a91f1d5. The gap is in getShowCreateInfo(), which this PR did not change. Its property list is the same on master and branch-4.x and has no entry for compute_group, while SHOW CREATE MATERIALIZED VIEW renders every MV property, so the MV half already round-trips. On master no job could declare a group and the statement never carried the implicit one, so a job recreated from it has always run in the recreating session's group. For a job that declares its group, though, losing the declaration on export defeats the point of declaring it. The statement now includes compute_group from getDeclaredComputeGroup(), which follows ALTER, and only when a group is declared. Jobs without a declaration keep their current output (RoutineLoadJobTest#testGetShowCreateInfo is unchanged); writing the effective group instead would turn an inherited group into a pinned one when the statement is replayed. RoutineLoadJobTest#testGetShowCreateInfoCarriesDeclaredComputeGroup checks the statement of a job that declares cg_a and is then altered to cg_b, and test_routine_load_compute_group now also checks SHOW CREATE ROUTINE LOAD after its ALTER.
The same list also leaves out workload_group and seven other properties CREATE accepts (send_batch_parallelism, load_to_single_tablet, partial_update_new_key_behavior, unique_key_update_mode, enclose, escape, empty_field_as_null). That predates this PR and affects every routine load job, so it is left to a separate change.
| } | ||
| } | ||
|
|
||
| private void applyComputeGroup(ConnectContext ctx) { |
There was a problem hiding this comment.
[P2] Apply the MV binding to incremental dry-run contexts
Keeping this initialization private to MTMVTask misses RefreshMTMVCommand.dryRunRefresh(), which creates an internal context through MTMVPlanUtil.createMTMVContext() and executes the delta query directly. That initializer copies workload_group but never the new compute_group. For an IVM MV pinned to B with a workload group existing only in B, while admin defaults to A, a nonempty REFRESH MATERIALIZED VIEW mv INCREMENTAL WITH DRY RUN looks up that workload group in A and fails although real refresh succeeds in B. Apply the declaration in shared MV context initialization or explicitly cover this foreground execution path too.
There was a problem hiding this comment.
Confirmed, and fixed in 45cda592e43. dryRunRefresh() builds its own context with MTMVPlanUtil.createMTMVContext(), which copies the MV's workload_group but sets no compute group. A non-empty delta was therefore planned and run wherever admin resolves (its default group, otherwise the first eligible one), and NereidsCoordinator.enqueue looked the workload group up in that group; an empty delta is answered by the FE and never got that far. The dry run now builds its context in createDryRunConnectContext(), which binds the declared group and re-checks it before the delta query is built, as the refresh paths in MTMVTask do. EXPLAIN REFRESH binds it to its planning context as well, matching what the task does for its own planning context; it runs nothing, so it does not re-check. An MV without a declaration keeps its current resolution. Because the dry run only reads, the rows it returned were correct; the problem was that it ran, and resolved its workload group, in a group the MV is not bound to. RefreshMTMVCommandTest covers the binding of both contexts (testDryRunContextBindsDeclaredComputeGroup, testExplainContextBindsDeclaredComputeGroup), a dropped declared group (testDryRunRejectsDroppedDeclaredComputeGroup), and an MV without a declaration (testContextsWithoutDeclarationKeepImplicitComputeGroup).
The binding is not in createMTMVContext(). That initializer also builds the contexts that only analyze the MV query (the rewrite cache built on user queries, and the relation check after a base table change); a re-check there would make a dropped group fail those user queries, and MTMVTask applies the declared group and then the session's on top of it anyway. branch-4.x has neither the dry run nor EXPLAIN REFRESH, so both paths exist only on master.
TPC-H: Total hot run time: 28886 ms |
TPC-DS: Total hot run time: 151166 ms |
ClickBench: Total hot run time: 24.39 s |
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: None Related PR: apache#69749 Problem Summary: apache#69749 lets CREATE and ALTER ROUTINE LOAD declare `compute_group`, stored in the job's `jobProperties`. SHOW CREATE ROUTINE LOAD prints `RoutineLoadJob.getShowCreateInfo()`, whose fixed property list had no entry for it. A job created in a session on cg_a with `"compute_group" = "cg_b"`, or moved to cg_b by ALTER, was therefore exported without its declaration. Recreating it from that statement in a session on cg_a gave a job that declares nothing: it snapshots cg_a from the session and runs every task there, with no error. SHOW CREATE MATERIALIZED VIEW renders every MV property, so an MV's declaration already round-trips. The statement now writes `compute_group` from `getDeclaredComputeGroup()`, which ALTER updates, between `timezone` and `exec_mem_limit`, and only when a group is declared. A job without a declaration keeps its current output; that includes every job outside cloud mode, where the property is rejected. The effective group (`getCloudCluster()`) is deliberately not written: replaying the statement would turn a group inherited from the creating session into a pinned one. ### Release note None ### Check List (For Author) - Test: Unit Test - New `RoutineLoadJobTest#testGetShowCreateInfoCarriesDeclaredComputeGroup`: a job that declares cg_a and is then altered to cg_b prints the statement `testGetShowCreateInfo` expects plus `"compute_group" = "cg_b"`; it fails with the new line removed. `testGetShowCreateInfo` (no declaration) is unchanged and passes. - With the next commit, 29 related FE test classes (`RoutineLoadJobTest`, `ShowCreateRoutineLoadCommandTest`, `RoutineLoadComputeGroupTest`, `KafkaRoutineLoadJobTest`, ...), 558 tests, pass with Checkstyle (`mvn -pl :fe-core -am test`) - `test_routine_load_compute_group` now also checks SHOW CREATE ROUTINE LOAD after its ALTER; not run locally, it needs a cloud cluster and Kafka - Behavior changed: Yes. SHOW CREATE ROUTINE LOAD of a job that declares `compute_group` now includes it - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pute group ### What problem does this PR solve? Issue Number: None Related PR: apache#69749 Problem Summary: `REFRESH MATERIALIZED VIEW mv INCREMENTAL WITH DRY RUN` is not forwarded: it runs on the current FE in an internal context built by `MTMVPlanUtil.createMTMVContext()`, which copies the MV's `workload_group` but sets no compute group. Its delta query was therefore planned and run wherever `admin` resolves (its default compute group, otherwise the first eligible one), while since apache#69749 every real refresh of an MV that declares `compute_group` runs in that group (`MTMVTask#applyComputeGroup`). Take an MV that declares cg_b with `workload_group = wg_b`, a workload group created only for cg_b, while admin resolves to cg_a: a dry run over a non-empty delta failed because the coordinator looked wg_b up in cg_a, although the refresh itself succeeds. Without a workload group the delta ran on cg_a's backends, and still ran after cg_b was dropped, when a real refresh fails with "Compute group 'cg_b' not found.". This follows from the code and was not reproduced on a cluster. EXPLAIN REFRESH planned in admin's group too, so its plan assigned backends from there. The dry run now builds its context in `createDryRunConnectContext()`, which binds the declared group and re-checks it with `ComputeGroupBindingUtil#checkComputeGroupBeforeTask`, as the refresh paths of `MTMVTask` do, before the delta query is parsed and planned. `createExplainConnectContext()` binds the declared group without the re-check, as `MTMVTask` does for its planning context, since EXPLAIN runs nothing. An MV that declares no group keeps the current resolution on both. The binding stays out of the shared `createMTMVContext()`, which also builds the analysis-only contexts of the rewrite cache and of the relation check, where a re-check would make a dropped group fail user queries. `test_mtmv_compute_group` is hardened as well. Its check after re-declaring the same group refreshed an unchanged base table, which is NOT_REFRESH and returns before a compute group is applied, so the task records none, and it read the first row of an unordered tasks() result. It now inserts a row first and checks the task that refresh created, found by id. ### Release note None ### Check List (For Author) - Test: Unit Test - `RefreshMTMVCommandTest` gains four tests: the dry run context and the EXPLAIN REFRESH context bind the declared group, the dry run rejects a declared group that no longer exists, and an MV without a declaration leaves both contexts unbound. Each of three mutations makes its test fail: no binding on the dry run context, no binding on the EXPLAIN REFRESH context, and no re-check in the dry run. - With the previous commit, 29 related FE test classes, 558 tests, pass with Checkstyle (`mvn -pl :fe-core -am test`), among them `RefreshMTMVCommandTest`, `ExplainRefreshMTMVCommandTest` and `NereidsParserTest` - `test_mtmv_compute_group` changed but not run locally: its compute group part needs a cloud cluster - Behavior changed: Yes. For an MV that declares `compute_group`, REFRESH ... WITH DRY RUN and EXPLAIN REFRESH run in that group, and the dry run fails when the group no longer exists - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the second review round in 7611a91 and 45cda59.
From the summary (item 10), RoutineLoadJobTest gains one test and RefreshMTMVCommandTest four. The SHOW CREATE test and three of the four (the binding of each context, the dry run's re-check) are each red without their fix; the fourth covers an MV without a declaration. 29 related FE test classes, 558 tests, pass with Checkstyle. |
There was a problem hiding this comment.
Static review completed against the verified exact head aaff9b57e30daa5c4c6b469d3af322e93fbe4297 and the authoritative merge-base diff. Three P2 findings are included inline: ALTER can persist a routine-load workload group that does not exist in the new compute-group namespace; a paused Kafka job can retain a retryable reason and repeatedly auto-resume after its owner's group USAGE is revoked; and concurrent MV ALTER can make a refresh validate one compute group while executing with another.
The existing inline threads were revalidated at this head and their fixes remain effective; none still applies as a P0/P1 finding, so existing_blocking_comment_ids is empty. The FE locking, exception/error-code, Nereids context, replay, cloud/non-cloud guard, workload-group, and backend-selection checkpoints were covered. The review focus file contains no additional user focus. Tests and regression suites were inspected but not executed, as required by the static-only contract.
Review coverage converged after two bounded rounds with no unresolved candidates.
Review converged after 2 round(s).
| String computeGroup = jobProperties.get(CreateRoutineLoadInfo.COMPUTE_GROUP); | ||
| if (!StringUtil.isEmpty(computeGroup)) { | ||
| // Unlike workload group, the compute group can be fully validated right here, so do | ||
| // it now instead of letting a bad name pause the whole job on the next task. |
There was a problem hiding this comment.
[P2] Validate the retained workload group in the target compute group
ALTER ROUTINE LOAD ... PROPERTIES("compute_group" = "cg_b") validates cg_b but leaves an existing workload_group unchanged without checking its namespace. A job created with compute_group=cg_a, workload_group=wg_a (where wg_a exists only in cg_a) can therefore persist the pair after switching to cg_b; on resume, KafkaTaskInfo.rePlan() binds cg_b and WorkloadGroupMgr throws because wg_a is absent there, so every task is paused with CREATE_TASKS_ERR. Validate the effective workload group against the requested compute group before logging the ALTER, or reject the cross-namespace change.
There was a problem hiding this comment.
Fixed in c9d9730047e. An ALTER that sets compute_group now checks the job's workload group in that group, as CREATE does in the group it declares. KafkaRoutineLoadJob.modifyProperties resolves the workload group the job will have afterwards, which is the one the ALTER sets or else the one the job keeps. The lookup runs as the owner the tasks run as, on the statement thread before the job lock, and the ALTER is rejected when the group does not have that workload group. It is the lookup CREATE uses, now shared as CreateRoutineLoadInfo.checkWorkloadGroup. The AlterRoutineLoadCommand comment that said the workload group could not be checked at ALTER was wrong: the job is known there, and #50563 checked its workload group in the job's group in the old modifyProperties(AlterRoutineLoadStmt). The Nereids modifyProperties(AlterRoutineLoadCommand) from #49960 never carried that check, and #53490 removed the statement path. As a result, an ALTER that sets workload_group alone is still unchecked on master and branch-4.x. That predates this PR and is unchanged here, as is DROP WORKLOAD GROUP, which does not consider routine load jobs either. The new RoutineLoadComputeGroupTest#testAlterChecksWorkloadGroupInTheComputeGroupItDeclares covers three cases: a move that keeps a workload group the new group lacks, a move that names a workload group of the new group, and an ALTER without compute_group. It fails without the check, and also when the check runs in the job's current group.
| } | ||
|
|
||
| // For failures found by refreshKafkaPartitions(), which update() runs before it takes the job lock. | ||
| // The state is checked again under the lock: a STOP that completed in between must not be turned |
There was a problem hiding this comment.
[P2] Stop auto-resume when a paused job loses group USAGE
pauseIfNeedSchedule() only records the supplied reason when state == NEED_SCHEDULE. An already PAUSED Kafka job with a retryable reason and no custom partitions enters updateKafkaPartitions() through needAutoResume(), but a revoked owner USAGE throws CANNOT_RESUME_ERR while this helper does nothing because the state is still PAUSED. The old retryable pauseReason remains, so ScheduleRule keeps auto-resuming and repeating the failed check. Preserve the lock/STOP guard while recording the non-retryable reason (or otherwise disabling auto-resume) for this path.
There was a problem hiding this comment.
Fixed in c9d9730047e. Confirmed: auto resume runs the partition refresh while the job is still PAUSED, and pauseIfNeedSchedule() only pauses a NEED_SCHEDULE job (PAUSED -> PAUSED is rejected, on replay too), so a failing check there kept the old, retryable reason. This came with the refresh check in 4bf82ff4a9d. Before that commit, and on branch-4.x, which has no check in the refresh, the attempt reached the task scheduler's check and recorded CANNOT_RESUME_ERR. The refresh now skips the compute group check for an auto resume attempt. Once the metadata request succeeds, the attempt moves the job to NEED_SCHEDULE, and the check RoutineLoadTaskScheduler runs before the first task pauses the job with CANNOT_RESUME_ERR under the job lock, which stops auto resume. A manual RESUME leaves the job in NEED_SCHEDULE, so the refresh still runs the check then, and a dropped group is still reported as Compute group '<name>' not found.. The comments at the check and at pauseIfNeedSchedule() now state this contract. The new RoutineLoadComputeGroupTest#testAutoResumeLeavesRevokedUsageToTheTaskCheck drives refreshKafkaPartitions(true) for a paused job whose owner lost USAGE, then schedules its first task. It checks that the job ends up paused with CANNOT_RESUME_ERR and is not auto resumed, and it fails when the refresh runs the check for the attempt.
| @@ -1680,6 +1684,34 @@ private void setupComputeGroup(ConnectContext ctx) { | |||
| } | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
[P2] Use one compute-group snapshot for binding and validation
setupComputeGroup() calls applyComputeGroup(), which reads mtmv.getComputeGroup() and installs that value in the context, then checkComputeGroupBeforeTask() reads the MV property again. ALTER MATERIALIZED VIEW can update the property between those two read locks, so a task can validate cg_b while its context remains bound to cg_a and the refresh executes there. Capture the declared/effective group once and use that snapshot for both context setup and the pre-task check (or validate the context's actual group).
There was a problem hiding this comment.
Fixed in c9d9730047e. The race is real but narrow, and its effect is smaller than described. The refresh ran in the group that was declared when its context was bound, which is a valid order for an ALTER that lands meanwhile. The only problem was that the re-check could look at the new group instead. That shows up when the old group is dropped at the same time: the refresh then failed later, at backend selection, with a generic error instead of Compute group '<name>' not found.. No data was affected. setupComputeGroup() now reads the declaration once, binds and records it, and re-checks that same value. Both refresh paths go through it, and the REFRESH ... WITH DRY RUN context already worked this way. The new MTMVComputeGroupTest#testRefreshContextChecksTheGroupItBinds makes the first read return a dropped group and any later read an existing one. It expects a single read and a check that fails for the bound group, and it fails when the check reads the property again.
TPC-DS: Total hot run time: 150987 ms |
ClickBench: Total hot run time: 24.37 s |
FE Regression Coverage ReportIncrement line coverage |
…nd MV refresh ### What problem does this PR solve? Issue Number: None Related PR: apache#69749 Problem Summary: Three gaps in the compute_group binding of routine load jobs and async MVs: 1. ALTER ROUTINE LOAD ... PROPERTIES("compute_group" = "cg_b") validated cg_b but not the job's workload_group there. A workload group lives in the namespace of a compute group, and every task looks the job's one up in the group the job runs in (KafkaTaskInfo#rePlan). A job created with compute_group = cg_a and workload_group = wg_a, where only cg_a has wg_a, accepted the move. Every task then failed with "Can not find workload group wg_a in compute croup cg_b", the job was paused with the retryable CREATE_TASKS_ERR, and auto resume repeated that indefinitely. CREATE already checks the workload group in the declared group. KafkaRoutineLoadJob#modifyProperties now checks, in the declared group, the workload group the job has after the ALTER (the one the ALTER sets, or else the one it keeps). It runs as the owner the tasks run as, on the statement thread before the job lock, with the lookup CREATE uses, now shared as CreateRoutineLoadInfo#checkWorkloadGroup. An ALTER that sets no compute_group is unchanged. 2. The Kafka partition refresh checks the declared compute group before its metadata request. Auto resume runs that refresh for a job that is still PAUSED, and there pauseIfNeedSchedule() does nothing: only a NEED_SCHEDULE job is paused, and PAUSED -> PAUSED is rejected. When the owner's USAGE on the group was revoked while the job was paused with a retryable reason, the check failed without a trace. The job kept its old reason, CANNOT_RESUME_ERR was never recorded, and ScheduleRule kept resuming it. The refresh now skips the check for an auto resume attempt. Once the metadata request succeeds, the attempt moves the job to NEED_SCHEDULE, and the check that RoutineLoadTaskScheduler runs before the first task pauses it with CANNOT_RESUME_ERR, which stops auto resume. After a manual RESUME the refresh still runs the check. 3. An MV refresh context bound the declared compute group from one read of the MV property and re-checked it from a second read. An ALTER MATERIALIZED VIEW between the two could have the check pass for the new group while the refresh ran in the old one. If the old group had been dropped, the refresh failed later with a generic no-backend error instead of "Compute group '<name>' not found.". setupComputeGroup() now reads the declaration once and binds, records and re-checks that value, as the REFRESH ... WITH DRY RUN context already does. No data was affected. ### Release note None ### Check List (For Author) - Test: Unit Test - RoutineLoadComputeGroupTest#testAlterChecksWorkloadGroupInTheComputeGroupItDeclares - RoutineLoadComputeGroupTest#testAutoResumeLeavesRevokedUsageToTheTaskCheck - MTMVComputeGroupTest#testRefreshContextChecksTheGroupItBinds; MTMVComputeGroupTest#testRefreshRechecksOnlyTheDeclaredComputeGroup now goes through setupComputeGroup() - 17 routine load, ALTER/SHOW CREATE, binding util and MV test classes (244 tests) pass, fe-core Checkstyle clean; checking in the job's current group or as the ALTER user, re-checking on auto resume, and re-reading the MV property before the check each turn only the matching new test red - Behavior changed: Yes, within this PR's feature: an ALTER that moves a routine load job to a compute group without its workload group is rejected, and a paused job whose owner lost USAGE on its declared group is paused with CANNOT_RESUME_ERR by its next auto resume attempt - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed review round 5 in
RoutineLoadComputeGroupTest gains two tests and MTMVComputeGroupTest one, each red without its fix; 17 related FE test classes, 244 tests, pass with Checkstyle. |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of all 25 changed files at the verified head c9d9730 found one new P2 issue: routine-load CREATE and ALTER validate and can auto-start a declared compute group before the existing table/job LOAD authorization gate, allowing a rejected DDL to mutate compute-group state. The inline comment is anchored to the activation call in CreateRoutineLoadInfo.
The MV/context sweep found no new issue beyond existing threads; current-head fixes for replay resource resolution, suspended-group activation, combined ALTER offsets, SHOW CREATE, dry-run binding, and the MV snapshot race were revalidated without duplicate comments. The scheduler sweep found no additional bug: queued tasks are checked before allocation and transaction begin, STOP races are fenced, PAUSED auto-resume is covered by its existing thread, and Kinesis creation remains non-cloud.
Required checkpoints were covered: locking and state transitions, CREATE/ALTER/replay lifecycle, effective compute-group propagation to backend and workload-group resolution, compatibility of existing property maps/edit logs, and parallel Kafka/Kinesis/MV paths. The review focus file contains no additional user focus. Tests and regression suites were inspected but not executed; no builds, benchmarks, cluster runs, or source edits were performed. The review converged after 2 rounds.
Review converged after 2 round(s).
| // The name is always adopted, because the workload group check below resolves in its | ||
| // namespace, but it is only validated when a user is actually declaring it. On the | ||
| // metadata load path the declared group may have been dropped long ago, and failing | ||
| // there would cancel the job permanently instead of pausing it - see isReplay. |
There was a problem hiding this comment.
[P2] Check routine-load authorization before activating the target group
The new compute_group branch calls activateDeclaredComputeGroup during statement validation. For CREATE, CreateRoutineLoadInfo.validate() runs before RoutineLoadManager.createRoutineLoadJob() checks table/database LOAD; for ALTER, AlterRoutineLoadCommand.validate() runs before alterRoutineLoadJob() calls checkPrivAndGetJob(). A user who lacks LOAD on the table/job can therefore wake a suspended compute group (or get a group-specific validation error) even though the DDL will be rejected. Move the side-effecting validation/activation after the existing LOAD authorization gate, or make it conditional on an already-authorized job.
There was a problem hiding this comment.
The order is as described, but the early step gives a user nothing they could not already do, so this PR keeps it. Before activateDeclaredComputeGroup runs, validateDeclaredComputeGroup makes the same two checks, in the same order, as USE @<compute group>: the user must hold USAGE on the group, and the group must exist. USE @<compute group> then starts a suspended group in the same way (CloudEnv.changeCloudCluster: checkCloudClusterPriv, then waitForAutoStart), so anyone who passes these checks can already wake the group from a session. master also validates the declared workload_group, its privilege check included, in the same validate() before RoutineLoadManager checks LOAD. What remains is that a statement later denied LOAD may have started a group its user is allowed to use, and may report a group error before the LOAD error. Moving the activation behind the LOAD gate would split it from the validation into RoutineLoadManager and KafkaRoutineLoadJob (CREATE before checkCustomPartition, ALTER before convertOffset) without changing what a user is allowed to do.
TPC-H: Total hot run time: 28719 ms |
TPC-DS: Total hot run time: 151393 ms |
ClickBench: Total hot run time: 24.26 s |
|
run cloud_p0 |
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
…tity-only context The previous commit made CreateRoutineLoadInfo's temporary context inherit the session's narrowing (F-17). That context exists only to resolve the job's workload group by name for an existence check -- placement, not data access, and the narrowed session may use every group the person's own session may use (the placement rule of this series). Narrowing the lookup therefore changes nothing a user can observe, and master has since moved that lookup into a static helper that builds its own identity-only context (apache#69749), so the inherited form would only conflict on merge. The data-access site of F-17, the connector rewrite's INSERT ... SELECT task context, keeps inheritIdentityFrom. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…er the job lock ### What problem does this PR solve? Issue Number: None Related PR: apache#70130, apache#69749 Problem Summary: The previous commit checks the workload group an ALTER ROUTINE LOAD leaves the job with before the Kafka and the Kinesis job take their write lock, and RoutineLoadManager does not serialize two ALTERs of one job. Two sessions altering the same paused job at once could therefore each check a different combination and leave one that neither checked: for a job in cg_a with wg_old, which cg_a and cg_b both have, an ALTER moving it to cg_b and one setting wg_a, which only cg_a has, both passed, and the job was left with wg_a in cg_b, which fails every task with the retryable CREATE_TASKS_ERR. The check-before-lock placement came from apache#69749, so on master two concurrent moves, one of which also sets a workload group, already raced this way. It takes cloud mode: outside it no ALTER changes the namespace the workload group is looked up in. Both job types now check first thing under the write lock, before the PAUSED check, so the later of two concurrent ALTERs checks what the earlier one left. The Kafka offset conversion stays before the lock: it sends a metadata request through the group's backends and does not depend on the workload group. So an ALTER with Kafka offsets and a workload group it cannot use now converts the offsets before it fails. The lookup is in memory (WorkloadGroupMgr, the access manager and the compute group names), and the same locked section already writes the edit log; no path takes the job lock while holding one of those. ### Release note None ### Check List (For Author) - Test: Unit Test - RoutineLoadComputeGroupTest#testConcurrentAltersCheckWhatTheEarlierOneLeft: the second ALTER starts from inside the first one's workload group lookup and waits for the job lock; with the check before the lock both were accepted and the job ended up with wg_a in cg_b, now the second fails with "Can not find workload group wg_a in compute croup cg_b" and the job keeps wg_old - 11 routine load test classes, 108 cases, pass - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…70118) ### What problem does this PR solve? Issue Number: None Related PR: #69749 Problem Summary: **In short.** `SHOW CREATE ROUTINE LOAD` prints the statement a job is recreated from, but a job recreated from it was not the same job: eight properties that CREATE accepts were never printed, a value with a quote or a backslash made the statement unparseable, `TEMPORARY` partitions turned into formal ones, and the comment was dropped. This PR makes the statement carry all of them. **Background.** Users recreate a routine load job from `SHOW CREATE ROUTINE LOAD` to rename it, to move it to another database, or to start it over after `STOP`; the statement prints the next Kafka offsets to consume for exactly that purpose. #69749 made the statement carry the `compute_group` a job declares, and found the rest of the list incomplete: `RoutineLoadJob.getShowCreateInfo()` writes the properties by hand and covered 17 of the 25 keys in `CreateRoutineLoadInfo.PROPERTIES_SET`. **The problem, and what it cost.** Measured on master `1aa505cb0be` with a unit test that creates jobs from SQL, recreates each one from its `SHOW CREATE` output and compares every property: | Job | Recreated from its SHOW CREATE output | |---|---| | CSV job with `enclose` `"`, `escape` `\`, `empty_field_as_null`, `send_batch_parallelism` = 3, `load_to_single_tablet`, `workload_group` | all six back to their defaults; quoted fields get split at the separators inside the quotes | | `unique_key_update_mode` = `UPDATE_FLEXIBLE_COLUMNS` with `partial_update_new_key_behavior` = `ERROR` | an `UPSERT` job with `APPEND` | | partial columns with `partial_update_new_key_behavior` = `ERROR` | `APPEND`: rows with new keys get inserted instead of failing the load | | JSON job with `jsonpaths` such as `["$.k1","$.v1"]` | the statement does not parse (`mismatched input '$'`) | | Kafka property whose value holds a `"`, such as a JAAS config | the statement does not parse | | `TEMPORARY PARTITION(tp1)` | printed as `PARTITION(tp1)`: `Partition tp1 does not exist` | | `COMMENT '...'` | gone | The jobs themselves keep running as declared; only the statement is wrong. The cost is a recreated job that silently loads differently (CSV parsing, partial update semantics, single-tablet loading, workload group, parallelism), or a statement that cannot be executed at all. **How this PR fixes it.** - `getShowCreateInfo()` prints the eight missing properties wherever leaving one out would give the recreated job a different value: `unique_key_update_mode` when it is not `UPSERT` (`partial_columns` alone cannot express `UPDATE_FLEXIBLE_COLUMNS`), `partial_update_new_key_behavior` when it is `ERROR` and a partial update mode is on (CREATE rejects the policy otherwise), `enclose` and `escape` when set, `empty_field_as_null` and `load_to_single_tablet` when true, and `workload_group` when set. `send_batch_parallelism` is always printed, as the timezone is, because CREATE otherwise takes it from the session variable of whoever recreates the job. The values are the ones the job runs with. - Keys and values are quoted with `SqlLiteralUtils.quoteStringLiteral`, which follows the session's `NO_BACKSLASH_ESCAPES` mode. A value without a quote or a backslash prints exactly as before; the column separator is left as it is, since it is kept as typed and already reads back. - `TEMPORARY PARTITION(...)` keeps `TEMPORARY`, and the job's comment is printed as `COMMENT "..."`. - `CREATE ROUTINE LOAD` reads its `COMMENT` with `SqlLiteralUtils.parseStringLiteral`, as table, view and column comments are read, so that a printed comment with a quote reads back unchanged. It used to strip the outer quotes only, so `COMMENT 'it''s'` was stored as `it''s`. For a job that uses none of these, the output gains a single line, `"send_batch_parallelism" = "<the job's value>"`. **Results.** | Job | Before | After | |---|---|---| | CSV job above | 6 properties and the comment lost | recreated with the same definition | | flexible partial update with `ERROR` | `UPSERT` / `APPEND` | same definition | | JSON job with `jsonpaths` | statement does not parse | same definition | | Kafka property with `"` and `\` | statement does not parse | same definition | | temporary partition | `Partition tp1 does not exist` | same definition | | `NO_BACKSLASH_ESCAPES` session, `enclose` `"`, `escape` `\` | both lost | same definition | The same test also found that a job's `max_filter_ratio`, `exec_mem_limit`, `send_batch_parallelism`, `load_to_single_tablet`, `enclose` and `escape` fall back to their defaults when a follower replays the job or the FE restarts, and that `ALTER ROUTINE LOAD` on `enclose` / `escape` does not reach the running job. That is a separate defect and will be fixed in its own PR; since this PR prints the values the job runs with, the statement follows that fix. branch-4.1 and branch-4.2 have the same gaps as master.
…mpute group (#70125) ### What problem does this PR solve? Issue Number: None Related PR: #69749, #62606 Problem Summary: **In short.** In cloud mode, `REFRESH MATERIALIZED VIEW ... INCREMENTAL WITH DRY RUN` and `EXPLAIN REFRESH MATERIALIZED VIEW` of an MV that declares no `compute_group` plan (and the dry run executes) in whatever compute group the `admin` user resolves to, not in the group of the session that issues them, where a manual `REFRESH` of the same MV runs. This PR binds both to the session's group, the way a manual REFRESH borrows it. **Background.** Both statements build an internal context with the admin identity and the MV's session variables (`MTMVPlanUtil.createMTMVContext`). Its `cloud_cluster` is empty, so it resolves a group the way admin would: admin's `default_compute_group`, else the first group admin may use. A manual REFRESH passes the session's group to its task instead (`MTMVJobManager#refreshMTMV`), and the task binds it when the MV declares none (`MTMVTask#applyComputeGroup`). #69749 made both statements honour a declared group; an MV that declares none, which is every MV created before #69749 and any MV created without the property, still goes to admin's group. Found while reviewing #69749 (thread 4401555648). **The problem, and what it cost.** Measured on master `856194ece0e` with a unit test in cloud mode: groups `cg_of_admin` and `cg_of_session`, the session on `USE @cg_of_session`, an MV that declares no group. | | manual REFRESH | dry run | EXPLAIN REFRESH | |---|---|---|---| | compute group | `cg_of_session` | `cg_of_admin` | `cg_of_admin` | - If the MV's `workload_group` exists only in the session's group (a workload group belongs to one compute group in cloud mode), the dry run fails with `Can not find workload group <wg> in compute croup cg_of_admin`, while the manual REFRESH succeeds. - Otherwise the dry run's delta query runs on the backends of another group, which the session's user may have no USAGE on, and starts that group if it is suspended. EXPLAIN REFRESH assigns backends there and can start it as well. Results are not affected: the dry run only reads, and storage is shared. **How this PR fixes it.** - The private lookup of the session's group in `MTMVJobManager` moves unchanged to `MTMVPlanUtil.getSessionComputeGroup()`, so a manual REFRESH, a dry run and EXPLAIN REFRESH resolve it with the same code. - `RefreshMTMVCommand` builds both internal contexts through one helper. It reads the session's group first, while the session is still the thread-local context, then binds the declared group, else the session's group, as `MTMVTask#applyComputeGroup` does. With neither, the context stays unbound and the existing resolution applies; outside cloud mode nothing changes. - As in `MTMVTask`, the dry run re-checks only a declared group, since the MV never chose the session's group. EXPLAIN REFRESH executes nothing and still checks nothing. **Results.** | MV | session | dry run / EXPLAIN REFRESH, before | after | |---|---|---|---| | declares no group | `USE @cg_of_session` | `cg_of_admin` | `cg_of_session`, as a manual REFRESH | | declares no group | resolves no group | admin's resolution | unchanged | | declares `cg_declared` | `USE @cg_of_session` | `cg_declared` | unchanged | | any, outside cloud mode | | no compute group | unchanged | branch-4.1 and branch-4.2 have neither statement.
… does (#70130) ### What problem does this PR solve? Issue Number: None Related PR: #69749, #50563, #49960, #53490 Problem Summary: **In short.** `ALTER ROUTINE LOAD ... PROPERTIES("workload_group" = "...")` accepts any name. `CREATE ROUTINE LOAD` checks the workload group as the job's owner in the compute group the job runs in, and every task of the job resolves it the same way, so a name that is missing there, or that the owner may not use, is accepted by the ALTER and then fails every task. This PR checks it at ALTER time, as CREATE does. **Background.** A workload group belongs to a compute group in cloud mode, and to the owner's resource group outside it. Each task looks the job's workload group up as the owner in the job's compute group (`KafkaTaskInfo#rePlan`, `KinesisTaskInfo#rePlan`). Kafka jobs used to check an altered workload group (#50563, in the `AlterRoutineLoadStmt` path). The Nereids `AlterRoutineLoadCommand` path (#49960) did not carry that check, and #53490 removed the old path. #69749 brought a check back only for an ALTER that moves a Kafka job to another compute group. The comment that delays the check ("we can only get clusterId when alter job") is no longer true: the ALTER has the job at hand. Found while reviewing #69749 (thread 4415694066). **The problem, and what it cost.** Measured on master `856194ece0e` with unit tests; each ALTER below goes straight to the PAUSED check, so on a paused job it is accepted and written to the edit log: | Job | ALTER | Result | |---|---|---| | Kafka, cloud mode, runs in `cg_a` | `"workload_group" = "wg_b"`, which only `cg_b` has | accepted | | Kafka, outside cloud mode | a workload group that does not exist | accepted | | Kinesis, outside cloud mode | a workload group that does not exist | accepted | After RESUME, the first task of each such job fails with `Can not find workload group ...`, and the job pauses with `CREATE_TASKS_ERR`. That code is retryable, so auto resume repeats this with backoff (20 s up to 300 s) for as long as the job exists. The ALTER that caused it reported success. **How this PR fixes it.** - `RoutineLoadJob#checkAlteredWorkloadGroup`, called by the Kafka and the Kinesis job first thing under the job's write lock, before the PAUSED check, so that of two ALTERs of one job running at once the later one checks what the earlier one left. The Kafka offset conversion stays before the lock, so an ALTER with Kafka offsets and a workload group it cannot use converts the offsets before it fails. When an ALTER sets a workload group or declares a compute group, it checks the resulting workload group in the resulting compute group, with `CreateRoutineLoadInfo.checkWorkloadGroup` as CREATE does. The resulting workload group is the one the ALTER sets, else the one the job keeps; the resulting compute group is the one the ALTER declares, else the job's. The check runs as the owner, because the tasks run as the owner. A rejected ALTER changes nothing and writes no edit log. The replay of an ALTER does not go through this path. - The Kafka job's check for a move to another compute group (#69749) is replaced by this one; it checks the same thing for that ALTER. - The comments in `AlterRoutineLoadCommand` that pointed at the old place, or at the old reason for not checking, are corrected. - `test_routine_load_alter` altered a job to `alter_routine_load_group` without creating the group: it worked only because the ALTER did not check, or because `test_routine_load` happened to create the group first. It now creates the group, the way `test_routine_load` does, and checks that an ALTER to a missing group fails. **Results.** | ALTER | Before | After | |---|---|---| | a workload group the job's compute group has, and the owner may use | accepted | accepted | | a workload group the job's compute group lacks, or the owner may not use | accepted, then every task fails and auto resume loops | rejected, e.g. `Can not find workload group wg_b in compute croup cg_a`; the job is unchanged | | a compute group, keeping a workload group the new group lacks (Kafka) | rejected (#69749) | rejected | | neither property | not checked | not checked | branch-4.1 and branch-4.2 have the same gap.
What problem does this PR solve?
Issue Number: None
Related PR: #67010
Problem Summary:
In short. This ports branch-4.1 #67010 (also on branch-4.2) to master: CREATE / ALTER ROUTINE LOAD and async materialized views accept a
compute_groupproperty in cloud mode, so a background job can be pinned to a compute group instead of inheriting one implicitly, and the declared group is re-checked before every task.Background. In cloud mode every query runs in a compute group. A routine load job snapshots the compute group of the session that created it, and an async MV has no compute group of its own: an automatic refresh runs wherever
adminresolves to, and a manualREFRESHborrows the group of the session that triggered it.The problem, and what it cost. On master nothing lets a user choose where a background job runs:
"compute_group" = "cg_etl"on CREATE or ALTER pins itadminresolves toREFRESHREFRESH ... WITH DRY RUNandEXPLAIN REFRESHadminresolves tocompute_groupin non-cloud mode, orDEFAULTFour smaller defects go with it on master: CREATE ROUTINE LOAD checks the
workload_groupin the session's compute group namespace although the job runs elsewhere;RoutineLoadJob.plan()builds the sink's plan context from the raw snapshot field rather than the effective group; and the statement re-parsed when a routine load job is loaded from metadata resolves itsworkload_groupagain. On image load that lookup runs before any workload group exists (PersistMetaModulesreadsroutineLoadJobsbeforeworkloadGroups, and until then the Env holds the emptyWorkloadGroupMgrits constructor created), and a failure there sets the job to the finalCANCELLED, so a job created withworkload_groupis cancelled by the first restart that loads it from an image (shown by a unit test with an empty manager, not reproduced on a cluster). And when the partition refresh fails, whichRoutineLoadJob.update()runs before it takes the job lock, it pauses a NEED_SCHEDULE job without that lock, so a STOP that completes in between is turned back into PAUSED on the master while replay keeps STOPPED (Kafka metadata failures and Kinesis shard listing alike).How this PR fixes it. Same design as #67010:
RoutineLoadJob.jobProperties,MTMV.mvProperties) and written only when declared: no new persisted field, journal type or meta version, and an absent key keeps today's implicit resolution, which is also how a later binding model tells a pinned group from an inherited one.DEFAULTis rejected because that model reserves it for "follow the owner's default group"; non-cloud mode rejects the property, andConnectContext.setCloudCluster()is a no-op outside cloud mode.ComputeGroupBindingUtilholds the two checks:validateDeclaredComputeGroupat CREATE / ALTER (USAGE for the current user, then existence, the order ofUSE @<group>), andcheckComputeGroupBeforeTaskbefore every task, against the identity the task runs as. A missing group keepsINTERNAL_ERR, so a routine load job is paused retryably and resumes by itself when the group comes back (a group scaled to zero backends leaves the cluster map and returns); a revoked USAGE usesCANNOT_RESUME_ERR, soScheduleRuledoes not keep resuming it.RoutineLoadJob.getCloudCluster(), which backend selection, the task sent to BE, the workload group namespace, the plan context and theComputeGroupcolumn of SHOW ROUTINE LOAD all use, so every Kafka job follows it (Kinesis routine load is rejected in cloud mode, the only mode with compute groups).RoutineLoadTaskSchedulerre-checks the declared group before backend allocation and beforebeginTxn. The partition refresh of a Kafka job runs the same check before its metadata request, which goes through the group's backends and, after RESUME, comes before any task: a dropped group pauses the job withCompute group '<name>' not found.instead ofno alive backendsand advice to check the Kafka properties. An auto resume attempt skips that check: it refreshes a job that is still PAUSED, which cannot be paused again with another reason, so the attempt goes on to NEED_SCHEDULE and the check before the first task records a revoked USAGE asCANNOT_RESUME_ERR. Every pause from the refresh (this check, a failed Kafka metadata request, a failed Kinesis shard listing) goes throughRoutineLoadJob.pauseIfNeedSchedule(), which takes the job lock and checks NEED_SCHEDULE again, so it cannot undo a concurrent STOP. CREATE validates the workload group in the declared group's namespace. An ALTER that setscompute_groupvalidates in that group the workload group the job has afterwards (the one the ALTER sets, or else the one it keeps), as the owner the tasks run as. An ALTER that setsworkload_groupalone is still not checked, as on master. The statement re-parsed at metadata load adopts both names without resolving either, because a failure there would cancel the job for good: the declared group can have been dropped (and the stored statement is the CREATE one, so a job moved to another group by ALTER still names its old group there), and on image load the workload groups are not loaded yet.USE @<group>does (waitForAutoStart), on the statement thread before any job lock, since they send their Kafka metadata requests (custom partitions, offset conversion) through that group's backends. An ALTER that setscompute_groupandkafka_offsetstogether converts the offsets in the group it declares, because the job's current group may be gone. The scheduler keeps resolving backends by name without starting a suspended group, as master does for every cloud routine load job.compute_groupwhen the job declares one. It writes the declared value, which follows ALTER, and never the effective group, so a job recreated from the statement is pinned the same way, and the statement of a job that declares nothing does not pin the group it inherited. SHOW CREATE MATERIALIZED VIEW already renders every MV property.REFRESH ... INCREMENTAL WITH DRY RUNruns outside the task, on the current FE, in a context of its own: it binds the declared group to that context and re-checks it the same way.EXPLAIN REFRESHbinds it to its planning context without the re-check, as the task does for its own. Neither borrows the session's group for an MV that declares none, as on master.Where master differs from branch-4.x:
MTMVTasksplits a refresh into the IVM incremental path and the partition / complete path, each with its own context.applyComputeGroup()is factored out of the existingsetupComputeGroup(), and the re-check runs in both paths.KinesisRoutineLoadJobrejects cloud mode, so a Kinesis job never has a compute group to declare.REFRESH ... WITH DRY RUN(from master's IVM) andEXPLAIN REFRESHexist only on master. Both build their context inRefreshMTMVCommandinstead ofMTMVTask, so that command binds the declared group itself, and the dry run re-checks it.compute_group; branch-4.x does not.compute_groupchecks the job's workload group in that group; branch-4.x does not, so there a job moved to a group without its workload group fails every task.MTMVJobalways creates a task with a context, and master'ssetupComputeGroup()already relies on it.MTMVComputeGroupTestgains two cases for the master-only parts:setupComputeGroup()applies and records the declared group, and the MV re-check fails for a declared group that no longer exists while it ignores the session's group.RefreshMTMVCommandTestgains four for the dry run and EXPLAIN REFRESH contexts, andRoutineLoadJobTestone for SHOW CREATE ROUTINE LOAD.ConnectContextTest.testThreadInfoUsesSessionCloudCluster([fix](fe) Report active compute group in processlist #68111) sets the session variable directly instead of callingConnectContext.setCloudCluster(), which is now a no-op outside cloud mode; branch-4.x made the same change when it picked [fix](fe) Report active compute group in processlist #68111 ([fix](fe) Report active compute group in processlist (#68111) #68948).Results. The unit tests below pass locally. The regression suites need a cloud cluster and were not run locally:
test_routine_load_compute_groupandtest_mtmv_compute_groupare p0 suites, so P0 runs their non-cloud rejection checks and cloud_p0 runs them in cloud mode (the routine load part also needs Kafka);test_compute_group_binding_isolationis a docker suite that starts its own two-group cloud cluster.Release note
Support declaring
compute_groupon routine load jobs and async materialized views in cloud mode, so a background job can be pinned to a specific compute group instead of inheriting one implicitly.Fix routine load jobs created with
workload_groupbeing cancelled when the FE restarts from an image.Check List (For Author)
cloud_p0/multi_cluster/test_compute_group_binding_isolation,load_p0/routine_load/test_routine_load_compute_group,mtmv_p0/test_mtmv_compute_group, carried over from branch-4.1: [feature](cloud) Support declaring compute_group on routine load and async MV #67010. Two checks go further than branch-4.1: [feature](cloud) Support declaring compute_group on routine load and async MV #67010:test_routine_load_compute_groupalso checks SHOW CREATE ROUTINE LOAD after its ALTER and produces to a topic of its own on each run, so its exact row count holds when it reruns against the same broker, andtest_mtmv_compute_groupchecks the task that the refresh after the same-value ALTER created, with a new row to load. Not run locally (cloud cluster and Kafka needed).RoutineLoadComputeGroupTest,MTMVComputeGroupTest,ComputeGroupBindingUtilTest,RefreshMTMVCommandTest,RoutineLoadJobTest, plus the existing tests of the changed classes and of everysetCloudCluster()caller (mvn -pl :fe-core -am test, Checkstyle included): 29 classes, 558 tests, all pass. After the third review round:RoutineLoadComputeGroupTest,ComputeGroupBindingUtilTest,RoutineLoadJobTest,RoutineLoadSchedulerTest,KafkaRoutineLoadJobTest,RoutineLoadTaskSchedulerTest,RoutineLoadManagerTest,RoutineLoadBackendSelectionTestandRoutineLoadJobPersistenceTest: 9 classes, 98 tests, all pass. After the fourth:RoutineLoadComputeGroupTest,RoutineLoadJobTest,KafkaRoutineLoadJobTest,KinesisRoutineLoadJobTest,RoutineLoadManagerTest,RoutineLoadSchedulerTestandRoutineLoadTaskSchedulerTest: 7 classes, 93 tests, all pass. After the fifth:RoutineLoadComputeGroupTest,RoutineLoadJobTest,KafkaRoutineLoadJobTest,KinesisRoutineLoadJobTest,RoutineLoadManagerTest,RoutineLoadSchedulerTest,RoutineLoadTaskSchedulerTest,RoutineLoadBackendSelectionTest,RoutineLoadJobPersistenceTest,AlterRoutineLoadCommandTest,AlterRoutineLoadOperationLogTest,ShowCreateRoutineLoadCommandTest,ComputeGroupBindingUtilTest,MTMVComputeGroupTest,MTMVTaskTest,RefreshMTMVCommandTestandExplainRefreshMTMVCommandTest: 17 classes, 244 tests, all pass, and fe-core Checkstyle is clean.setupComputeGroup()ignoring the declaration, the MV re-check looking at the session's group, the routine load scheduler skipping the re-check, andsetCloudCluster()recording a group outside cloud mode. Four more cover the review fixes: resolving the workload group on the metadata load path (two tests: a dropped declared group, and an emptyWorkloadGroupMgras on image load), CREATE or ALTER not starting the declared group, and a combined ALTER converting its offsets in the job's old group. The second review round adds four more: SHOW CREATE ROUTINE LOAD leaving the declared group out, the dry run context or the EXPLAIN REFRESH context not binding it, and the dry run skipping its re-check. The third round adds one: the Kafka partition refresh skipping the check. The fourth adds one: the refresh pausing without the job lock, which makes all three new STOP-race tests fail. The fifth adds three: an ALTER that setscompute_groupchecking the workload group in the job's current group instead of the declared one, the partition refresh running the compute group check for an auto resume attempt, and the MV re-check reading the declaration a second time.compute_group: such an MV refreshes in its declared group even for a manualREFRESHfrom another group, and itsREFRESH ... WITH DRY RUNandEXPLAIN REFRESHrun and plan there too; such a job fails its next task when the group is dropped or its owner's USAGE on it is revoked, and such an MV's dry run fails when the group is dropped; an ALTER that moves a job to a group that lacks its workload group is rejected; SHOW CREATE ROUTINE LOAD of such a job includescompute_group. Outside cloud modeConnectContext.setCloudCluster()no longer records a value. Metadata load no longer resolves a routine load job'sworkload_group, so such a job is no longer cancelled on restart; CREATE / ALTER ROUTINE LOAD wait for a suspended declared group to start, asUSE @<group>does.compute_groupproperty of routine load and async MV: cloud mode only for now,DEFAULTreserved.Check List (For Reviewer who merge this PR)